Replication run started: 2025-08-20T13:45:23.347301
Root: /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication
Hardware: cpu
Sections: ['analysis']
Continue on error: True

-- Running section: analysis (scripts found: 12)

--- Starting analysis/01_section_3.py at 2025-08-20T13:45:23.348044 ---


>>> Running: analysis/01_section_3.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/01_section_3.py
Human-LLM agreement percent: 0.925
Human-LLM Cohen's Kappa: 0.85
Human-LLM disagree count: 30
Human-LLM reasonable disagreements: 16.0
LLM error count: 14.0
Total augmented hypotheses: 2834
Test set size: 15366
Validation set size: 15036
Training set size: 171289
Validation set documents with hypotheses in the training set: 5094
Validation set documents with hypotheses not in the training set: 9942
Test set documents with hypotheses in the training set: 0

>>> Exit code: 0
--- Finished analysis/01_section_3.py at 2025-08-20T13:45:26.722246 (rc=0) ---

--- Starting analysis/02_section_5.py at 2025-08-20T13:45:26.722291 ---


>>> Running: analysis/02_section_5.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/02_section_5.py
Claude 3.5 Polistance Quote Tweets F1: 0.646
DEBATE Base Polistance Quote Tweets F1: 0.837
DEBATE Large Polistance Quote Tweets F1: 0.94

>>> Exit code: 0
--- Finished analysis/02_section_5.py at 2025-08-20T13:50:14.424395 (rc=0) ---

--- Starting analysis/03_section_6.py at 2025-08-20T13:50:14.424425 ---


>>> Running: analysis/03_section_6.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/03_section_6.py
Human benchmark: 0.87
Llama zero shot: 0.63
Llama 25 shot: 0.6

>>> Exit code: 0
--- Finished analysis/03_section_6.py at 2025-08-20T13:50:15.970065 (rc=0) ---

--- Starting analysis/05_section_7.py at 2025-08-20T13:50:15.970098 ---


>>> Running: analysis/05_section_7.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/05_section_7.py

>>> Exit code: 0
--- Finished analysis/05_section_7.py at 2025-08-20T13:50:17.267003 (rc=0) ---

--- Starting analysis/06_appendix_a.py at 2025-08-20T13:50:17.267038 ---


>>> Running: analysis/06_appendix_a.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/06_appendix_a.py

>>> Exit code: 0
--- Finished analysis/06_appendix_a.py at 2025-08-20T13:50:25.325845 (rc=0) ---

--- Starting analysis/07_appendix_b.py at 2025-08-20T13:50:25.325876 ---


>>> Running: analysis/07_appendix_b.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/07_appendix_b.py

>>> Exit code: 0
--- Finished analysis/07_appendix_b.py at 2025-08-20T13:55:16.172688 (rc=0) ---

--- Starting analysis/08_appendix_c.py at 2025-08-20T13:55:16.172726 ---


>>> Running: analysis/08_appendix_c.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/08_appendix_c.py

>>> Exit code: 0
--- Finished analysis/08_appendix_c.py at 2025-08-20T13:55:16.482691 (rc=0) ---

--- Starting analysis/09_appendix_d.py at 2025-08-20T13:55:16.482719 ---


>>> Running: analysis/09_appendix_d.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/09_appendix_d.py
/Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/09_appendix_d.py:251: SyntaxWarning: invalid escape sequence '\.'
  groups['group'] = groups['group'].replace('\.', '', regex = True)
/Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/09_appendix_d.py:506: SyntaxWarning: invalid escape sequence '\.'
  poldf['target'] = poldf['target'].replace('\.', '', regex = True)

>>> Exit code: 0
--- Finished analysis/09_appendix_d.py at 2025-08-20T13:55:17.769406 (rc=0) ---

--- Starting analysis/10_appendix_e.py at 2025-08-20T13:55:17.769435 ---


>>> Running: analysis/10_appendix_e.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/10_appendix_e.py

>>> Exit code: 0
--- Finished analysis/10_appendix_e.py at 2025-08-20T13:58:55.316053 (rc=0) ---

--- Starting analysis/11_appendix_f.py at 2025-08-20T13:58:55.316080 ---


>>> Running: analysis/11_appendix_f.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/11_appendix_f.py

>>> Exit code: 0
--- Finished analysis/11_appendix_f.py at 2025-08-20T13:58:56.736516 (rc=0) ---

--- Starting analysis/12_appendix_g.py at 2025-08-20T13:58:56.736544 ---


>>> Running: analysis/12_appendix_g.py
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/python /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/12_appendix_g.py
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/Skylion007/openwebtext/resolve/f3808c30e817981b845ec549c43e82bb467d8144/subsets/urlsf_subset02.tar
Retrying in 1s [Retry 1/5].
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/Skylion007/openwebtext/resolve/f3808c30e817981b845ec549c43e82bb467d8144/subsets/urlsf_subset02.tar
Retrying in 1s [Retry 1/5].
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/Skylion007/openwebtext/resolve/f3808c30e817981b845ec549c43e82bb467d8144/subsets/urlsf_subset02.tar
Retrying in 1s [Retry 1/5].
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/Skylion007/openwebtext/resolve/f3808c30e817981b845ec549c43e82bb467d8144/subsets/urlsf_subset02.tar
Retrying in 2s [Retry 2/5].
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/Skylion007/openwebtext/resolve/f3808c30e817981b845ec549c43e82bb467d8144/subsets/urlsf_subset02.tar
Retrying in 4s [Retry 3/5].
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/lucadiliello/STORIES/resolve/9585988079b67ffee934cd417f3edb23c3eb5401/data/train-00016-of-00069-cd8d1a164fe0d914.parquet
Retrying in 1s [Retry 1/5].
HTTP Error 504 thrown while requesting GET https://huggingface.co/datasets/lucadiliello/STORIES/resolve/9585988079b67ffee934cd417f3edb23c3eb5401/data/train-00016-of-00069-cd8d1a164fe0d914.parquet
Retrying in 2s [Retry 2/5].
Found 6822 substring matches in OpenWebText.
Found 151 substring matches in CC News.
Found 111 substring matches in OpenWebText.
Found 890 substring matches across all datasets.

>>> Exit code: 0
--- Finished analysis/12_appendix_g.py at 2025-08-20T15:32:09.558188 (rc=0) ---

--- Starting analysis/04_section_6_covid_replication.r at 2025-08-20T15:32:09.559288 ---


>>> Running: analysis/04_section_6_covid_replication.r
>>> Command: /Users/mb7336/miniforge3/envs/debate_analysis/bin/Rscript /Users/mb7336/Library/CloudStorage/OneDrive-TexasA&MUniversity/DEBATE/replication/analysis/04_section_6_covid_replication.r
Warning message:
package ‘MASS’ was built under R version 4.4.2 
There were 50 or more warnings (use warnings() to see the first 50)

>>> Exit code: 0
--- Finished analysis/04_section_6_covid_replication.r at 2025-08-20T15:44:02.335662 (rc=0) ---

Replication run finished: 2025-08-20T15:44:02.335714 (final_rc=0)
